Lectures 9 and 10 - Experiments and Experimental Design
Argument, Data, and Politics - POLS 3312
Invalid Date
Procedural
- Review of midterm
- Minor procedures update
- Questions on procedural issuesSubstantive
- Writing assignment (Monday)
- Recap of scientific method, cause and effect (Monday)
- Introduction to experiments
- Experimental design
- Lab and Field experiments
- Natural experiments, quasi-experiments, etc.Has anyone every discussed why this is done? Any thoughts?
Exam serves at least three functions
- The exam is a diagnostic tool
- The exam is a learning tool
- The exam is a credentialing tool Exam serves at least three functions
The functions require different things
- The diagnostic and learning aspects both require a broad variation in outcomes, a wide frequency distribution of raw scores to measure differences and similarities across the class.
- The credentialing aspect is why we adjust the scores.Why not use an actual normal distribution to adjust the scores?
Exam serves at least three functions
The functions require different things
- The diagnostic and learning aspects both require a broad variation in outcomes, a wide frequency distribution of raw scores to measure differences and similarities across the class.
- The credentialing aspect is why we adjust the scores.Why not use an actual normal distribution?
- Upper division class
- Distribution isn't usually normal
- A normal distribution would require some people to fail regardless of quality of work. It is possible everyone deserves the credential. It is possible though rarer that no one does. Raw scores
- 3 raw scores of 40
- 2 raw scores of 39.5
- 9 raw scores of 37.5 to 38.5
- 3 raw scores of 37
- 17 people got all the available points or more
- Average raw score was 29.16, a C+Adjusted scores
- Adjusted average 30.09 (81.3%, a B-)
- Top adjusted score was 41Biggest issues
- Not attempting written answers
- Written answers that were very shallow - "Because that's how we do it."
- Not reading the questions carefully
- Answers are posted. You may share them, but not with commercial websites including but not limited to Chegg or CourseHero.Class dismissal time1
- 10 minutes to the hour, University policy for all classes
- Most courses are 90 minute courses, so 80 minutes of class time
- If you must leave early, please do not leave during the final five minutes of classClass dismissal time1
- 10 minutes to the hour, do not leave during last five minutes of classWritten assignments
- If you must leave early and I have not collected them, you may leave them as you leave
- These are mostly learning tools, so I have generally given credit for good faith participation in the exercise
- Some require a bit moreDiscussion
Class dismissal time1
- 10 minutes to the hour, do not leave during last five minutes of classWritten assignments
Discussion
- Multiple comments from presenters about lack of discussion on articles
- Everyone needs to be reading these ahead of the class
- General class discussion - We are getting into less math intensive topics where more discussion is appropriate and expectedAny questions?
Which of the following is not a required element of an experiment?
Hint: Think about both of the February 28th readings.
Cause and effect
What is science?
Scientific method
Cause and effect
- What is a cause?What is science?
Scientific method
Cause and effect
- What is a cause?
- what is the general problem with finding true causes (includes two other problems)?What is science?
Scientific method
Cause and effect
- What is a cause?
- what is the general problem with finding true causes?
- What is the fundamental problem of causal inference? (Put differently: What is the hardest obstacle to determining true causes?)Cause and effect
- What is a cause?
- what is the general problem with finding true causes?
- What is the fundamental problem of causal inference? (Put differently: What is the main problem in determining true causes?)
- What is the problem in determining causes that statistical methods address?What is science?
Scientific method
Cause and effect: Three sets of problems
- General problem - causes are complicated
- Fundamental problem - we can't observe the "what if" scenario directly (the counterfactual)
- There is a random element to the world (statistical problem)What is science?
Scientific method
Cause and effect
What is science?
- Based on proof or evidence - concerns the world as it actually existsScientific method
Cause and effect
What is science?
- Based on proof or evidence
- About cause and effect - again the concern is with how the world actually worksScientific method
Cause and effect
What is science?
- Based on proof or evidence
- About cause and effect
- Scientific method involving falsifiable hypothesesScientific method
Cause and effect
What is science?
Scientific method
The steps in the scientific method parallel the order of the sections in a research paper.
Cause and effect
What is science?
Scientific method
The steps in the scientific method parallel the order of the sections in a research paper.
What is step 1?
Cause and effect
What is science?
Scientific method
1. Define a *research question*Cause and effect
What is science?
Scientific method
1. Define a *research question*In a paper we look two places. First, we review the current literature for answers to the question. Then we move to the next step a new theory and…
The theory section includes what? (Step Two)
Cause and effect
What is science?
Scientific method
1. Define a *research question*
2. Make predictions - *hypothesis*Once we have predictions, we need to do two things to test them. What is step 3?
Cause and effect
What is science?
Scientific method
1. Define a *research question*
2. Make predictions - *hypothesis*
3. Gather *data*Cause and effect
What is science?
Scientific method
1. Define a *research question*
2. Make predictions - *hypothesis*
3. Gather *data*After we have data, what do we do with it?
Cause and effect
What is science?
Scientific method
1. Define a *research question*
2. Make predictions - *hypothesis*
3. Gather *data*
4. Analyze the data to *test* the hypothesisAfter testing the hypotheses, what do we do?
Cause and effect
What is science?
Scientific method
1. Define a *research question*
2. Make predictions - *hypothesis*
3. Gather *data*
4. Analyze the data to *test* the hypothesis
5. Draw conclusionsIs drawing conclusions the end of the scientific process?
Cause and effect
- What is a cause?
- what is the general problem with finding true causes?
- What is the fundamental problem of causal inference? (Put differently: What is the main problem in determining true causes?)
- What is the problem in determining causes that statistical methods address?What is science?
- Based on proof or evidence
- About cause and effect
- Scientific method involving falsifiable hypothesesScientific method
1. Define a *research question*
2. Make predictions - *hypothesis*
3. Gather *data*
4. Analyze the data to *test* the hypothesis
5. Draw conclusions - Start again at step 1Experiments Chapter 7 from Empirical Methods in Political Science
Reading this will be of great use to you in understanding the topic for testing, future discussion in this class, and your own knowledge
In an experiment, we do it differently. We create the counterfactual. We create the “what if” scenario and the factual scenario and compare them.
How can we devise a way to create and compare the factual and counterfactual scenarios reliably?
Experiments are also known as randomized controlled trials (RCTs) or controlled studies.
The name “randomized controlled trial” tells us a lot about what an experiment is.
Controlled
- The researcher controls the manipulation (treatment or intervention)
- The subjects (people in the trial) are divided into control group and a treatment groupRandomized
An experiment has three vital components:
Lab experiments
Survey experiments
## Vignette example
From the previous example, what are some things we could “leave out” of the control vignette to study a particular treatement effect?
Field experiments
1
In a Get Out the Vote (GOTV) experiment, a typical design would:
Internal validity: Unbiased (not influenced by the researcher other than the manipulation)
- Design
- Content
- Analysis
Effective randomization is a major portion of internal validity
External validity: Generalizability
- To what populations can we generalize the results?
- To what settings can we generalize the results?
- To what times can we generalize the results?
Sample selection is a major portion of external validity
Author: Tom Hanna
Website: tomhanna.me
Do not submit to Chegg or similar websites
License: This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.</>
POLS3311, Spring 2024, Instructor: Tom Hanna